Fix incremental indexing reprocessing and document state handling - #112
Open
huytg2610 wants to merge 2 commits into
Open
Fix incremental indexing reprocessing and document state handling#112huytg2610 wants to merge 2 commits into
huytg2610 wants to merge 2 commits into
Conversation
Accented scripts have more than one valid encoding of the same text: "ệ" is
either U+1EC7 or "e" + U+0323 + U+0302. Both render identically, so the
difference is invisible, but the strings compare unequal and hash differently.
That matters because these strings become identity keys:
clean_text -> md5 -> doc_id (minirag.py:446)
clean_str -> entity_name -> graph node id
-> md5 -> ent-<hash> (vector id)
So the same document typed with a Vietnamese IME (NFC) and copied off the
macOS filesystem (NFD) produced two different doc_ids and was indexed twice,
and "PHÚ THỌ" written the two ways became two graph nodes that could never
merge — halving what a query could reach about one entity.
Pure-ASCII text has a single encoding, so NFC and NFD coincide and the bug
never fired for English. It is a no-op there, which is why it went unnoticed.
Normalise in clean_text and clean_str, the two funnels every such string
already passes through. Tests cover id stability across both forms, that
upper-casing before cleaning still converges (operate.py does this), and that
ASCII and the existing escape/control-character handling are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
The insertion pipeline rebuilt chunks from every PROCESSED document. Inserting an already-indexed document therefore reran entity extraction, and relationship weights could grow because edge merging accumulates weight. A duplicate insert now stages no document and makes no LLM extraction call.
Tests
Repository-wide test collection still needs optional PostgreSQL and Weaviate dependencies/services that are not present locally.